[https://nvbugs/6566734][test] Unwaive test_disaggregated_qwen3_32b_fp8 and accept both greedy completions by moraxu · Pull Request #18319 · NVIDIA/TensorRT-LLM

moraxu · 2026-08-27T20:45:03Z

Dev Engineer Review

Updated qwen3_32b_fp8 output validation to accept both known-good greedy completion starts.
Reused the existing any-of check pattern.
Removed the matching waiver entry from tests/integration/test_lists/waives.txt.
No public API, performance, or error-handling changes were identified.
Offline validation and local B200 validation passed.

QA Engineer Review

Modified test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8].
The test is covered in tests/integration/test_lists/waives.txt; its waiver entry was removed.
Verdict: sufficient.

Description

disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] failed in post-merge builds 2886 and 2887 on DGX_H200-8_GPUs-PyTorch-Post-Merge-1 and was auto-waived in #17340 (nvbugs/6566734).

Root cause: the test's output verification requires the exact substring "Asyncio in Python is a library" in the completion of the raw prompt "What are the benefits of using asyncio in Python?" (greedy, temperature=0, --ignore-eos, 100 tokens). That prompt sits on a near-tie between two greedy continuations:

mode A (historical passes): " Asyncio in Python is a library..." — answers the question
mode B: " I have read that it is used for asynchronous programming..." — continues the question; a coherent, on-topic raw completion

Both failing CI runs produced byte-identical mode-B text, and the other four prompts' completions were byte-identical to each other and healthy (no corruption; the "Berlin Berlin" canary absent). The 5 prompts are sent concurrently (asyncio.gather in disagg_client.py) into a 4x(ctx TP1) + 1x(gen TP4) cluster with Eagle3 speculation and FP8 KV cache, so batch composition — and therefore kernel-level numerics — is not run-invariant; a near-tied top-2 first token can legitimately flip.

The fix mirrors the existing precedent for deepseek_v3_lite (#6301, nvbugs/5390810): accept either known-good greedy completion via the check's existing any-of list mechanism, and remove the waive.

Test Coverage

disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] — unwaived by this PR (post-merge l0_dgx_h200).
Offline validation: the new check passes on the exact output.json contents captured from both failing CI runs (2886, 2887), while the old check reproduces the CI assertion on both.
Local 8xB200 validation (clean dual-arch wheel at 682fa40, the failing run's CI image): 3 server boots; the asyncio completion took the same mode-B start each time, byte-identical across boots. Fixed test: 2/2 PASSED (5:26 cold cache, 2:07 warm). A control run with the unmodified check also passed on B200 — but only because there the completion happens to wander into a sentence containing the literal phrase "Asyncio in Python is a library" ~30 tokens deep; on H200 the tail went elsewhere within the 100-token budget and CI failed. That tail-luck dependence is exactly the fragility this PR removes: the check now anchors on the two observed completion starts.
H200 validation on this PR: /bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" (as a multi-GPU stage on a PR this additionally requires the ci: full pre-merge approved label from an NVIDIA/trt-llm-ci-approvers member).

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

coderabbitai · 2026-08-27T20:48:29Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cc908a75-4e35-4a2c-aebf-8041932aa635

📥 Commits

Reviewing files that changed from the base of the PR and between 3d466db and 571a637.

📒 Files selected for processing (2)

tests/integration/defs/disaggregated/test_disaggregated.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

tests/integration/defs/disaggregated/test_disaggregated.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The Qwen3 32B FP8 disaggregated integration test now accepts two valid completion responses and is removed from the skipped-test list.

Changes

Qwen3 FP8 test enablement

Layer / File(s)	Summary
Completion validation and test activation `tests/integration/defs/disaggregated/test_disaggregated.py`, `tests/integration/test_lists/waives.txt`	The completion check accepts two valid asynchronous-programming responses and still requires the Berlin response. The test waiver entry is removed.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 571a6

This change broadens one test to accept two documented greedy completions and removes its waiver; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: bo-nv, nv-xtf

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required NVBugs and type format. It clearly states that the test is unwaived and that both valid greedy completions are accepted.
Description check	✅ Passed	The description includes complete Description, Test Coverage, and PR Checklist sections. It clearly explains the failure, root cause, fix, precedent, and validation results.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

moraxu · 2026-08-27T20:49:04Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-08-27T20:56:05Z

PR_Github #69801 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

tensorrt-cicd · 2026-08-27T22:49:43Z

PR_Github #69801 [ run ] completed with state SUCCESS. Commit: 20bcbb9
/LLM/main/L0_MergeRequest_PR pipeline #57095 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

moraxu · 2026-08-28T14:16:56Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-08-28T14:23:03Z

PR_Github #69982 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

tensorrt-cicd · 2026-08-29T14:22:55Z

PR_Github #69982 [ run ] completed with state ABORTED. Commit: 20bcbb9

Link to invocation

moraxu · 2026-08-31T18:59:55Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-08-31T19:06:27Z

PR_Github #70424 [ run ] triggered by Bot. Commit: 20bcbb9 Link to invocation

tensorrt-cicd · 2026-08-31T20:23:40Z

PR_Github #70424 [ run ] completed with state SUCCESS. Commit: 20bcbb9
/LLM/main/L0_MergeRequest_PR pipeline #57653 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…p8 and accept both greedy completions The qwen3_32b_fp8 disagg check required the exact substring "Asyncio in Python is a library" in the completion of a raw greedy prompt whose top-2 first tokens are near-tied: the model either answers the question or continues it. Post-merge builds 2886/2887 (DGX_H200) produced the question-continuation byte-identically and failed; the test was then waived. Accept either observed completion via the check's existing any-of list (same shape as PR NVIDIA#6301) and remove the waive. Signed-off-by: Michal Guzek <mguzek@nvidia.com>

coderabbitai · 2026-08-31T20:33:38Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

moraxu · 2026-09-01T04:32:43Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-09-01T04:39:41Z

PR_Github #70571 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

tensorrt-cicd · 2026-09-01T06:52:40Z

PR_Github #70571 [ run ] completed with state SUCCESS. Commit: 571a637
/LLM/main/L0_MergeRequest_PR pipeline #57783 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

moraxu · 2026-09-01T14:57:11Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-09-01T15:05:06Z

PR_Github #70712 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

tensorrt-cicd · 2026-09-01T16:56:07Z

PR_Github #70712 [ run ] completed with state SUCCESS. Commit: 571a637
/LLM/main/L0_MergeRequest_PR pipeline #57904 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

moraxu · 2026-09-02T16:02:05Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-09-02T16:12:50Z

PR_Github #71021 [ run ] triggered by Bot. Commit: 571a637 Link to invocation

moraxu requested a review from a team as a code owner August 27, 2026 20:45

moraxu requested review from bo-nv and nv-xtf August 27, 2026 20:45

github-actions Bot assigned moraxu Aug 27, 2026

moraxu force-pushed the user/mguzek/nvbug-6566734-qwen3-32b-disagg-flake branch from 20bcbb9 to 571a637 Compare August 31, 2026 20:33

schetlur-nv added the ci: full pre-merge approved label Sep 2, 2026

schetlur-nv approved these changes Sep 2, 2026

View reviewed changes

Uh oh!

Conversation

moraxu commented Aug 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

moraxu commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

moraxu commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 29, 2026

Uh oh!

moraxu commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

coderabbitai Bot commented Aug 31, 2026

Uh oh!

moraxu commented Sep 1, 2026

Uh oh!

tensorrt-cicd commented Sep 1, 2026

Uh oh!

tensorrt-cicd commented Sep 1, 2026

Uh oh!

moraxu commented Sep 1, 2026

Uh oh!

tensorrt-cicd commented Sep 1, 2026

Uh oh!

tensorrt-cicd commented Sep 1, 2026

Uh oh!

moraxu commented Sep 2, 2026

Uh oh!

tensorrt-cicd commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

moraxu commented Aug 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 27, 2026 •

edited

Loading